Accuracy of Entrustment-Based Assessment: Implications for Programs and Patients.

Background Although entrustment-supervision ratings are more intuitive compared to other rating scales, it is not known whether their use accurately assesses the appropriateness of care provided by a resident. Objective To determine the frequency of incorrect entrustment ratings assigned by faculty and whether accuracy of an entrustment-supervision scale differed by resident performance when the scripted resident performance level is known. Methods Faculty participants rated standardized residents in 10 videos using a 4-point entrustment-supervision scale. We calculated the frequency of rating a resident incorrectly. We performed generalizability (G) and decision (D) studies for all 10 cases (768 ratings) and repeated the analysis using only cases with an entrustment score of 2. Results The mean score by 77 raters for all videos was 2.87 (SD=0.86) with a mean of 2.37 (SD=0.72), 3.11 (SD=0.67) and 3.78 (SD=0.43) for the scripted levels of 2, 3, and 4. Faculty ratings differed from the scripted score for 331of 768 (43%) ratings. Most errors were ratings higher than the scripted score (223, 67%). G studies estimated the variance proportions of rater and case to be 4.99% and 54.29%. D studies estimated that 3 raters would need to watch 10 cases. The variance proportion of rater was 8.5% when the analysis was restricted to level 2 entrustment, requiring 15 raters to watch 5 cases. Conclusions Participants underestimated residents' potential need for greater supervision. Overall agreement between raters and scripted scores were low.

Lifelong Learning Indicators in Medical Students After a Novel Communication Skills Session.

CONTEXT: A growth mindset and mastery approach have gained attention as useful learning orientations in medical education, however few studies of interventions to foster these orientations exist. OBJECTIVES: We sought to discover whether a communication skills session on delivering serious news could foster a communication growth mindset and/or a mastery approach in medical students. METHODS: This was an interventional survey study of third-year medical students before and after a session on delivering serious news. Students were administered a communication mindset survey before and after the session; achievement goal and learning environment surveys were administered after the session. Chi-square tests were used to assess the difference in pre and post mindsets. Logistic regression was used to determine the odds of achieving a mastery approach with pre- and post-communication growth mindset as the independent variables. RESULTS: Students' communication growth mindset increased from 79% (n = 186) before the intervention to 92% (n = 142) after the intervention. Achievement goal analysis demonstrated that 64% (n = 91) of students had a mastery approach, 14% (n = 20) had a performance approach and 22% (n = 32) had an avoidant approach. Ninety-nine percent (n = 151) felt the session provided a safe learning environment. The odds of having a mastery approach correlated with both pre and post-intervention growth mindset, with post-session growth mindset having the strongest correlation. CONCLUSIONS: A novel communication skills session on delivering serious news fostered a communication growth mindset in third year medical students. Most students exhibited a mastery approach to learning; this approach was more likely when they had a growth mindset.

Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation.

BACKGROUND: While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. OBJECTIVE: To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). DESIGN: Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs). SUBJECTS: Participating faculty in longitudinal rater training. MAIN MEASURES: Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). KEY RESULTS: Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (ß 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender. CONCLUSIONS: Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

Evaluation of a Palliative Care Longitudinal Curriculum for Medical Students Using the Context-Input-Process-Product Model.

Palliative care (PC) longitudinal curricula are increasingly being recognized as important in Undergraduate Medical Education (UME). They are however, not yet commonplace, and where they do exist may be implemented without a systematic, prospective approach to curriculum evaluation. This paper describes an implementation of a new longitudinal curriculum at the Perelman School of Medicine (PSOM) at the University of Pennsylvania. We used the Context Input Process Product (CIPP) model, a holistic evaluation model, to assess the local environment, design the curriculum, guide the improvement process, and evaluate outcomes. Comprehensive models such as CIPP provide a more robust approach to curriculum evaluation than outcomes-only models and may be of use to other programs who are implementing new curricula or improving upon existing programs.

Preclerkship Medical Students' Use of Third-Party Learning Resources.

Importance: Undergraduate medical education increasingly relies on asynchronous, virtual learning; and medical educators have observed students engaging in self-directed learning outside of their institutional curriculum using widely available third-party resources. If medical educators better understand how students are learning, they may uncover novel opportunities to improve preclerkship education. Objective: To explore how and why preclerkship medical students use third-party learning resources. Design, Setting, and Participants: This qualitative study recruited second-year medical students from 7 public and private allopathic US medical schools and conducted 7 virtual focus groups (1 per institution) from September 2022 to January 2023, exploring how and why students use third-party resources. Data were iteratively analyzed in parallel with focus groups using constructivist grounded theory methodology. Data analysis was performed from October 2022 to February 2023. Results: Fifty-eight second-year US medical students who had used a third-party resource at least once participated; 36 (61%) identified as women; 13 (23%) identified as Asian, 6 (11%) as Black, 30 (53%) as White, 6 (11%) as multiracial, and 4 (7%) as other; 6 (10%) identified as Hispanic, Latino, or Spanish origin, and 52 (90%) identified as non-Hispanic, Latino, or Spanish origin; 48 (83%) were aged 23 to 25 years. Participants described engaging in a cyclical process of deciding whether and how to use third-party resources. Four broad themes were identified: (1) hearing about resources, (2) selecting resources, (3) using resources, and (4) tensions and possible solutions. Participants largely heard about third-party resources from peers and turned to resources out of dissatisfaction with some aspect of their medical school curriculum. Students used resources in various ways that were user-dependent and context-dependent. Participants endorsed multiple benefits over their in-house curricula, particularly efficiency, clarity, and concision. Tensions included navigating resource drawbacks and the perception of an antagonistic relationship between medical schools and third-party resources. Participants suggested that medical schools examine the resources, recommend specific ones, integrate them into the curriculum, and subsidize their cost. Conclusions and Relevance: In this qualitative study of preclerkship medical student use of third-party resources, participants perceived that the resources had numerous benefits for learning and suggested that medical schools should more formally acknowledge and integrate their use.

Association of applicant demographic factors with medical school acceptance.

BACKGROUND: Medical school acceptance rates in the United States (US) have been lower for applicants who identify as Underrepresented-in-Medicine (UiM) compared to non-UiM applicants. The gap between UiM and no-UiM groups is narrowing in recent years. Less well-studied are associations of acceptance decisions with family income and parental education. This study's purpose is to evaluate the relationships between medical school acceptance and family income, parental education status, racial/ethnic background, Grade Point Average (GPA), Medical College Admission Test (MCAT) score, and participation in extracurricular activities. METHODS: This is a cross-sectional study of first-time US medical school applicants between 2017 and 2020. Acceptance rates for first-time applicants were calculated for first-generation (FG), low-income (LI), and UiM applicants. Associations of these attributes with MCAT scores, science GPAs, and seven categories of extracurricular activities were evaluated. Regression analyses estimated associations between acceptance to medical school with all variables with and without interaction terms (FG*URM, LI*URM, FG*LI). RESULTS: The overall acceptance rate for first-time applicants from 2017-2020 was 45.3%. The acceptance rates among FG, LI and UiM applicants were 37.9%, 39.6% and 44.2%, respectively. In univariable logistic regression analyses, acceptance was negatively associated with being FG (OR: 0.68, CI: 0.67-0.70), LI (OR: 0.70, CI: 0.69-0.72), and UiM (OR: 0.95, CI: 0.93-0.97). In multivariable regression, acceptance was most strongly associated with science GPA (OR: 7.15, CI: 6.78-7.54 for the highest quintile) and UiM (OR: 5.56, CI: 5.48-5.93) status and MCAT score (OR: 1.19, CI: 1.18-1.19), FG (OR: 1.14, CI: 1.10-1.18), and most extracurricular activities. Including interaction terms revealed a negative association between acceptance and LI (OR:0.90, CI: 0.87-0.94) and FG was no longer significant (OR:1.10, CI:0.96-1.08). CONCLUSIONS: Collectively these results suggest medical school admissions committees may be relying on holistic admission practices. While MCAT and GPA scores continue to predict acceptance, individuals from racially and ethnically UiM backgrounds have favorable odds of acceptance when controlling for MCAT and GPA. However, these positive associations were not seen for low-income and first-generation applicants. Additional preparation for college and the MCAT for these latter groups may help further diversify the medical profession.

The medical humanities at United States medical schools: a mixed method analysis of publicly assessable information on 31 schools.

INTRODUCTION: There have been increasing efforts to integrate the arts and humanities into medical education, particularly during undergraduate medical education (UME). Previous studies, however, have focused on courses and curricular programming without rigorous characterization of the associated paracurricular environment or infrastructure enabling or facilitating these offerings. METHODS: To assess opportunities for students to engage the arts and humanities during their medical education as well as the institutional resources to support those opportunities, we developed the Humanities and Arts Programming Scale (HARPS): an 18-point scale involving eight sub-domains (Infrastructure, Curricular Opportunities, Extracurricular Engagement, Opportunities for Immersion, Faculty Engagement, Staff Support, Student Groups, and Scholarship). This scale was used to evaluate the top-31 ranked United States medical schools as determined by US News and World Report's (USWNR) Medical School Research Rankings using information derived from public-facing, online information. RESULTS: Mean cumulative HARPS score was 11.26, with a median score of 12, a standard deviation of 4.32 and a score range of 3-17. Neither USWNR ranking nor private/public institution status were associated with the cumulative score (p = 0.121, p = 0.739). 52% of institutions surveyed had a humanities-focused center/division with more than 70% of the schools having significant (> 5) faculty engaged in the medical humanities. 65% of schools offered 10 or more paracurricular medical humanities events annually, while 68% of the institutions had more than 5 medical humanities student organizations. While elective, non-credit courses are available, only 3 schools required instruction in the arts and humanities, and comprehensive immersive experiences in the medical humanities were present in only 29% of the schools. CONCLUSIONS: Although there is a significant presence of the medical humanities in UME, there is a need for integration of the arts and humanities into required UME curricula and into immersive pathways for engaging the medical humanities.

Early Engagement in HIV Research: Evaluation of the Penn CFAR Scholars Program Aimed at Increasing Diversity of the HIV/AIDS Workforce.

BACKGROUND: Demographic diversity is not represented in the HIV/AIDS workforce. Engagement of underrepresented trainees as early as high school may address this disparity. METHODS: We established the Penn Center for AIDS Research (CFAR) Scholars Program, a mentored research experience for underrepresented minority (URM) trainees spanning educational stages from high school to medical school. The program provides participants with tailored educational programming, professional skill building, and mentored research experiences. We conducted qualitative interviews with scholar, mentor, and leadership groups to evaluate the program's impact. RESULTS: Eleven participants were selected to partake in 1 of 5 existing mentored research programs as CFAR scholars. Scholars attended an 8-week HIV Seminar Series that covered concepts in the basic, clinical, behavioral, and community-based HIV/AIDS research. Program evaluation revealed that scholars' knowledge of HIV pathophysiology and community impact increased because of these seminars. In addition, they developed tangible skills in literature review, bench techniques, qualitative assessment, data analysis, and professional network building. Scholars reported improved academic self-efficacy and achieved greater career goal clarity. Areas for improvement included clarification of mentor-mentee roles, expectations for longitudinal mentorship, and long-term engagement between scholars. Financial stressors, lack of social capital, and structural racism were identified as barriers to success for URM trainees. CONCLUSION: The Penn CFAR Scholars Program is a novel mentored research program that successfully engaged URM trainees from early educational stages. Barriers and facilitators to sustained efforts of diversifying the HIV/AIDS workforce were identified and will inform future program planning.

Natural Language Processing of Learners' Evaluations of Attendings to Identify Professionalism Lapses.

Unprofessional faculty behaviors negatively impact the well-being of trainees yet are infrequently reported through established reporting systems. Manual review of narrative faculty evaluations provides an additional avenue for identifying unprofessional behavior but is time- and resource-intensive, and therefore of limited value for identifying and remediating faculty with professionalism concerns. Natural language processing (NLP) techniques may provide a mechanism for streamlining manual review processes to identify faculty professionalism lapses. In this retrospective cohort study of 15,432 narrative evaluations of medical faculty by medical trainees, we identified professionalism lapses using automated analysis of the text of faculty evaluations. We used multiple NLP approaches to develop and validate several classification models, which were evaluated primarily based on the positive predictive value (PPV) and secondarily by their calibration. A NLP-model using sentiment analysis (quantifying subjectivity of the text) in combination with key words (using the ensemble technique) had the best performance overall with a PPV of 49% (CI 38%-59%). These findings highlight how NLP can be used to screen narrative evaluations of faculty to identify unprofessional faculty behaviors. Incorporation of NLP into faculty review workflows enables a more focused manual review of comments, providing a supplemental mechanism to identify faculty professionalism lapses.

Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial.

PURPOSE: Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy. METHOD: This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance. RESULTS: Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments. CONCLUSIONS: Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.

The Context of "Confidence": Analyzing the Term Confidence in Resident Evaluations.

BACKGROUND: Despite similar performance metrics, women medical trainees routinely self-assess their own skills lower than men. The phenomenon of a "confidence gap" between genders, where women report lower self-confidence independent of actual ability or competency, may have an important interaction with gender differences in assessment. Identifying whether there are gender-based differences in how confidence is mentioned in written evaluations is a necessary step to understand the interaction between evaluation and the gender-based confidence gap. OBJECTIVE: To analyze faculty evaluations of internal medicine (IM) residents for gender-based patterns in the use of iterations of "confidence." DESIGN: We performed a retrospective cohort study of all inpatient faculty evaluations of University of Pennsylvania IM residents from 2018 to 2021. We performed n-gram text-mining to identify evaluations containing the terms "confident," "confidence," or "confidently." We performed univariable and multivariable logistic regression to determine the association between resident gender and references to confidence (including comments reflecting too little confidence), adjusting for faculty gender, post-graduate year (PGY), numeric rating, and service. SUBJECTS: University of Pennsylvania IM residents from 2018 to 2021. KEY RESULTS: There were 5416 evaluations of IM residents (165 women [51%], 156 men [49%]) submitted by 356 faculty members (149 women [51%]), of which 7.1 % (n=356) contained references to confidence. There was a significant positive association between the mention of confidence and women resident gender (OR 1.54, CI 1.23-1.92; p<0.001), which persisted after adjustment for faculty gender, numeric rating, and PGY level. Eighty evaluations of the cohort explicitly mentioned the resident having "too little confidence," which was also associated with women resident gender (OR 1.66, CI 1.05-2.62; p=0.031). CONCLUSION: Narrative evaluations of women residents were more likely to contain references to confidence, after adjustment for numerical score, PGY level, and faculty gender, which may perpetuate the gender-based confidence gap, introduce bias, and ultimately impact professional identity development.

The Impact of Trainee and Standardized Patient Race and Gender on Internal Medicine Resident Communication Assessment Scores.

BACKGROUND: Standardized patient (SP) encounters are commonly used to assess communication skills in medical training. The impact of SP and resident demographics on the standardized communication ratings in residents has not been evaluated. OBJECTIVE: To examine the impact of gender and race on SP assessments of internal medicine (IM) residents' communication skills during postgraduate year (PGY) 1. METHODS: We performed a retrospective cohort study of all SP assessments of IM PGY-1 residents for a standardized communication exercise from 2012 to 2018. We performed descriptive analyses of numeric communication SP ratings by gender, race, and age (for residents and SPs). A generalized estimating equation model, clustered on individual SP, was used to determine the association of gender (among SP and residents) with communication ratings. A secondary analysis was performed to determine the impact of residents and SP racial concordance in communication scores. RESULTS: There were 1356 SP assessments of 379 IM residents (199 male residents [53%] and 178 female residents [47%]). There were significant differences in average numeric communication rating (mean 3.40 vs 3.34, P = .009) by gender of resident, with higher scores in female residents. There were no significant interactions between SP and resident gender across the communication domains. There were no significant interactions noted with racial concordance between interns and SPs. CONCLUSIONS: Our data demonstrate an association of resident gender on ratings in standardized communication exercises, across multiple communication skills. There was not an interaction impact for gender or racial concordance between SPs and interns.

A Novel Ambulatory Curriculum for Pulmonary and Critical Care Fellowship Training.

Background: Dedicated ambulatory training during pulmonary and critical care medicine (PCCM) fellowships is often limited. A novel 2-year longitudinal outpatient pulmonary fellowship curriculum was previously developed, piloted, and studied. The exportability and potential impact of this ambulatory curriculum on PCCM fellowship training nationally is not known. Objective: We aim to understand the current state of ambulatory training in PCCM fellowships and the impact of a standardized outpatient curriculum on fellows' ambulatory knowledge and competency. Methods: Nineteen programs participated in the study from 2017 to 2019. Six programs received the first year of content, seven programs received the entire 2-year curriculum, and seven programs served as a control. Fellows, faculty, and program directors (PDs) completed a series of surveys assessing satisfaction with ambulatory education and the curriculum. Fellows completed a series of medical knowledge inventories, and programs submitted in-training exam scores. Results: A total of 221 fellows (39%) and 17 PDs (89%) completed the precurriculum surveys, and 38 (12%) fellows and 10 (53%) PDs completed postcurriculum surveys. Before curriculum implementation, only 34.4% of fellows rated the quality of their ambulatory education as good or outstanding compared with 57.9% at the end of the study. Eighty-five percent of faculty and 89% of PDs rated the curriculum as good or excellent. Faculty believed that the teaching scripts were easy to use (78.4%), were factually accurate (86.3%), and provided high-yield information (82.1%). The majority of PDs indicated that the curriculum positively impacted patient care (78%) and fulfilled an unmet educational need (100%), and most planned to continue the curriculum after the study (78%). Feedback surrounded the need for updated content based on recently published guidelines and studies. Conclusion: The curriculum is a standardized and feasible way to address a previously unmet need in PCCM fellowship education. PDs rated the curriculum highly and most plan to continue it in the future. Our limited data set suggests that the curriculum was well received by fellows and faculty and positively impacted perceptions of ambulatory education and preparedness for independent practice. Future study with a larger sample of fellows is needed to better understand the generalizability of these findings.

Peer Discussion Decreases Practice Intensity and Increases Certainty in Clinical Decision-Making Among Internal Medicine Residents.

BACKGROUND: Team-based decision-making has been shown to reduce diagnostic error, increase clinical certainty, and decrease adverse events. OBJECTIVE: This study aimed to assess the effect of peer discussion on resident practice intensity (PI) and clinical certainty (CC). METHODS: A vignette-based instrument was adapted to measure PI, defined as the likelihood of ordering additional diagnostic tests, consultations or empiric treatment, and CC. Internal medicine residents at 7 programs in the Philadelphia area from April 2018 to June 2019 were eligible for inclusion in the study. Participants formed groups and completed each item of the instrument individually and as a group with time for peer discussion in between individual and group responses. Predicted group PI and CC scores were compared with measured group PI and CC scores, respectively, using paired t testing. RESULTS: Sixty-nine groups participated in the study (response rate 34%, average group size 2.88). The measured group PI score (2.29, SD = 0.23) was significantly lower than the predicted group PI score (2.33, SD = 0.22) with a mean difference of 0.04 (SD = 0.10; 95% CI 0.02-0.07; P = .0002). The measured group CC score (0.493, SD = 0.164) was significantly higher than the predicted group CC score (0.475, SD = 0.136) with a mean difference of 0.018 (SD = 0.073; 95% CI 0.0006-0.0356; P = .022). CONCLUSIONS: In this multicenter study of resident PI, peer discussion reduced PI and increased CC more than would be expected from averaging group members' individual scores.

Assessment of Gender-based Qualitative Differences within Trainee Evaluations of Faculty.

Rationale: Recent studies of trainee evaluations of medical faculty have demonstrated subtle gender-based word choice differences. However, it is not known if this manifests in major contextual differences in written comments.Objectives: To characterize qualitative differences in narrative evaluations of female and male pulmonary and critical care medicine faculty.Methods: We performed a single-center, retrospective cohort analysis of trainee narrative evaluations of pulmonary and critical care medicine faculty at the Hospital of the University of Pennsylvania, written from 2015 to 2016. Directive and summative content analysis was performed by four raters. Major contextual themes were identified using constant comparative techniques. Thematic differences based on faculty gender were identified, with statistical significance determined using χ2 analysis (P < 0.05). Effect sizes were calculated using Cramér's V.Results: A total of 1,216 total narrative evaluations were analyzed, representing 62 faculty members (17 women and 45 men), with analysis continued for the full cohort beyond achieving saturation of themes. Five overarching themes emerged: teaching skills, clinical skills, supervision, interpersonal and communication skills, and leadership skills. Within subthemes, we found no significant gender differences in reference to general teaching skills, learning environment, enthusiasm for teaching, or interpersonal concern. We identified subtheme differences between male and female faculty evaluations in regard to mentions of learner autonomy, clinical learning environment, humor, and motivating the trainee for patient care. Although the mention of constructive criticisms did not differ between genders, constructive criticisms of female faculty were more likely to mention demeanor (P = 0.06).Conclusions: Our data suggest minimal thematic differences in trainee narrative evaluations of male and female pulmonary and critical care medicine faculty. However, we noted several subtheme contextual differences between male and female faculty, which warrants further investigation.